Development of Dynamic Image Recognition System for Hand Sign Language into Audio and Visual Output using Artificial Neural Networks

Abstract

This system provides a simple solution to a complex problem faced by deaf and dumb community. This paper visualizes the development web application of a real-time image recognition system whose main aim is to convert or translate the hand signs used in sign language in textual or visual and audio form output by using artificial neural networks. This system used computer vision to see signs as an input for processing and makes accurate predictions by using deep learning methods in real-time. By using artificial neural networks (ANN), the system achieves better performance in predicting or recognizing hand signs. The system makes communication between people using sign language and people who don’t know sign language effective by providing output as an audio and texted / visual, which even helps people in both ways. The project aims only to provide the society a solution by helping people who use sign language to express themselves to make it more effective and give them an ability to speak by using the technologies.

Introduction

I. INTRODUCTION

For deaf or signers, hand sign language plays an important role by becoming a bridge of communication for them, it also provides them a way to express themselves by using hand signs. However, for non-signers or the people who don’t understand sign language it is very difficult for them to understand. To provide solution to this problem, this study introduces with and web application that provides functionality of dynamic sign recognition system which is able to convert or translate the input hand signs of hand sign language into audio and visual outputs.

By specially using artificial neural networks for the system over convolutional neural networks, recurrent neural networks, for more accurate hand signs prediction.

The main aim of the system is to provide (Specially for deaf / signers) an accessibility of this web application to every individual which will enhance the communication and will give them an ability to express themselves more effectively and they can stand with everyone in the surrounding as an equal. There are about 14% of the worlds population who can speak or have problems while speaking and 14% of the worlds population is a large number so, we have provided a concept that can become a helping hand to the hand sign users by converting their hand signs in text and speech, which will make them feel like they are speaking or simply it gives them an ability to speak and express themselves more effectively.

II. METHODOLOGY

The system uses various technologies that helped in building a efficient and effective model for sign language recognition. Following figure presents or explains the flowchart of the system working and how it predicts hand signs and also below there is the explanation of all modules and libraries that are used for building the model.

It explains how image is captured using feature extraction by media pipe library of python programming then it is compared to already trained datasets which helps the models predict the hand signs which has more similarities and less errors and the printed label or output is converted in audio output.

A. Flow Chart:

Firstly the camera, OpenCV library in python is used due to which we were able to integrate with web cam which allows to system to capture performed hand signs of sign language. After capturing the image there are particular co-ordinated occurred on hand by using Media Pipe which helps system to predict hand sign by comparing the image points with already stored data set. There are about 6 layers of neural networks in the system which helps system predict the output more accurately. Once prediction is made the dedicated label is shown and the texted label is converted in the speech.

B. Modules Used for Implementation:

1. OpenCV: For Camera or web cam integration from which image is captured in the model.
2. Media Pipe: for feature extractions and for identifying hand co-ordinates to system for sign prediction.
3. TensorFlow: For creating model and its neural networks, in which there are nodes which acts as or can be said as mimic as a human brain in which there are neuron which transfer the data from one node to another it is also said as forward and backward propagation for the error solving and get the data with less error.
4. Different algorithms are used for the system to work efficiently and the algorithm is chosen on the bases of requirement.
5. web-speech API: This converts the predicted output in speech which enhances the communication for the people.

III. IMPLEMENTATION

A. Web Application Interface

The Web application provides the interactive user interface to its users for performing hand signs. The web application is made by keeping an aim in mind that it should be accessible to every individual who need our system. It has a try now button, after clicking it renders to new window and opens the camera on which users can perform the hand signs and the output can be taken as per the hand signs performed. The web application users interface can seen in fig. 1 and fig. 2.

Above Table. 3 contains the example of few sentences that can be created using the labels of signs in Table. 2 which are given as an example to make sentences using the above sentences and labels.

A. Validation Standards

Validating a hand sign recognition system using AI involves assessing its performance and ensuring its accuracy, robustness, and reliability.

Here are some standard validation points:

1. Range:  Hand sign recognition system hand a range for performing hand signs which lies between (15 cm to 1 m). Within this the system can accurately analyze and predict performed desired hand sign.
2. Number of hands: Hand sign recognition system can accurately recognize hand signs performed by one or two hands. Maximum 2 hand can be predicted accurately by the system.
3. Light area: Person performing hand signs should be performing the hand signs in the lighted area which will help the system to track hand signs more accurately. Dark areas should be avoided while performing hand signs.
4. Number of hand signs: Currently we have trained the model about 25 hand signs which can be utilized for making various sentences.
5. Web application: Hand sign recognition system using AI is deployed in a web application which is accessible to the user through internet.

By following these validation standards, you can ensure the reliability and effectiveness of an AI-based hand sign recognition system.

IV. RESULTS

In real time images are captures from the camera by the system thwn the captured images are given to algorithm or model for prediction or processing. The media pipe library of the python is used for accurate fearure extraction on hand co-ordinates. The captured co-ordinates are then compared with the already trained co-ordinates dataset for every label or word. Then the most compaired or similar co-ordinated label value will be predicted. The predicted sign will be visualized with the text and the text will be converted in to the speech.

V. FUTURE SCOPE

The Sign Language Recognition System using AI has laid a foundation for future advancements and applications. As technology continues to evolve, there are several exciting avenues for exploration and improvement in the field of sign language recognition.

1. Multi-Modal Integration: Future systems can explore the integration of multiple modalities, combining visual information with additional sensory inputs like facial expressions and contextual cues. This holistic approach can enhance the richness and accuracy of sign language interpretation.
2. Enhanced Gesture Vocabulary: Expanding the gesture vocabulary covered by the system can lead to more comprehensive communication. Research and development efforts can focus on capturing a broader range of signs, including regional variations and specialized signs in specific domains
3. Edge Computing and IoT Integration: Optimizing the system for edge computing and integration with Internet of Things (IoT) devices can enhance accessibility in various environments. Real-time, on-device processing can reduce latency and improve the system's applicability in different scenarios.
4. Cross-Cultural Adaptability: Adapting the system to recognize and accommodate different sign language variations worldwide enhances its cross-cultural applicability. Collaborative efforts can involve the creation of extensive datasets representing diverse signing styles.
5. Continuous Learning and Adaptation: Implementing mechanisms for continuous learning and adaptation will ensure the system stays relevant over time. This involves the ability to update the model with new signs, expressions, and linguistic nuances.

Conclusion

We conclude, that the development of the dynamic sign recognition system for hand signs language which represents such development as an advancement in technology. By Utilizing ANN (Artificial Neural Networks), the system’s performance is enhanced for real-time sign recognition and prediction of the hand sign. The audio and visual output enhances the communication for every individual with problem, by this it provides a real-time interactive environment to the needy users. In future more works or research could be done on this, new signs could be added, user interface could be enhanced. For making people express themselves more effectively and communication would be enriched.

References

[1] Ahmed B, Abdu H. G, Adel A, Muhammed A. (2023) has conducted an experiment on “Deep Learning in Sign Language Recognition: A Hybrid Approach for Recognition of Static and Dynamic Signs”, (Volume 11), (issue 17), Department of computer science.URL: https://doi.org/10.3390/math11173729 [2] Medhini P, Prasad H, Sai D, Shivam T. (2022) has conducted an experiment on “Sign Language Conversion to Text to Speech” (Volume 9), (Issue 7), Department of Computer Science and Engineering, Dr. Ambedkar Institute of Technology, Bangalore, 560056. URL: https://www.jetir.org>papers [3] Vilma V, E J Honesty P, G Maniknand, Ms. Hemalata S. (2024) has conducted an experiment on “Sign Language Detection and Recognition Using Media Pipe and Deep learning Algorithms” (Volume 11), (Issue 2), International Journal of Scientific Research in Science and Technology. URL: https:// ijsrts.com/index.php/home/article/view/IJSRST5241123 [4] Jean M, Trong N, Huu H. (2013) has conducted an experiment on “Static Hand Gesture Recognition Using Artificial Neural Networks” (Volume 1), (Issue 1), Journal of Image and Graphics. URL: https://joig.1.1.34-38 [5] Pigou L, Dieleman S, Kindermans J, Schrauwen B. (2015) “Sign Language Recognition Using Convolutional Neural Networks”. (Volume 8925) (issue 40) Lecture Notes in Computer Science,. Springer, Cham. URL: https://link.springer.com/chapter/10.007/978-3-319-16178-5_40 [6] Zaki M.M., Shaheen S. (2011) “Sign language recognition using a combination of new vision-based features” (Volume 32) (Issue 4). Links: [7] https://docs.opencv.org/2.4/doc/tutorials/imgproc/gausian_median_blur_bilateral_filter/gausian_median_blur_bilateral_filter.html [8] http://www-i6.informatik.rwth-aachen.de/~dreuw/database.php Git-hub Repository: [9] Daeshpande3.github.io “A Beginner Guide to Understanding Convolutional Neural Networks” Books: [10] \"Machine Learning for Sign Language Recognition\" by A. Smith (2021): Explores machine learning techniques specifically applied to sign language recognition, covering both traditional methods and recent advancements. [11] “Learn American Sign Language” by J. Guido (2015): Has in depth knowledge of the signs performed by signers for day-to-day communication, which helped us to study all the hand signs used were taught to the model.

Copyright

Copyright © 2024 Prof. M. C. Chavan , Prof. D. N. Chaudhari , Pratham J, Vinay M, Lalit C, Ashish T. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.